Known methods for obtaining an image (virtual viewpoint image) of a viewpoint different from a photographing position include tilt-shift shooting for shooting images with the optical axis moving, and a method for obtaining a free-viewpoint image by generating a distance map and combining a plurality of images of different viewpoints (multiple-viewpoint images) in accordance with the distance map. Known methods for obtaining multiple-viewpoint images include a method in which, as in Japanese Patent Laid-Open No. 2000-307947, an object is photographed a plurality of times in such a manner that areas including the object can overlap one another and resulting images are combined, and a method in which, as in Japanese Patent Laid-Open No. 2006-352539, a camera array is used. The latter is superior in terms of handheld shooting or the like.
Photographic systems for obtaining multi-viewpoint images using a camera array include a system in which, as in Japanese Patent Laid-Open No. 2006-352539, an object is photographed from all directions, and a system in which, as in ProFUSION25, which is manufactured by ViewPLUS Inc., images are photographed by a plurality of cameras arranged in a plane. In order to obtain an image of a viewpoint different from a photographing position, it is very important to acquire a distance map of a photographing area including an object. The arrangement of cameras forming a camera array greatly affects the accuracy of the distance map. As in Japanese Patent No. 3561446, a method in which cameras are arranged in front of and behind the optical axis has been proposed. Known methods for generating a distance map from multiple-viewpoint images include, for example, a plane sweep method.
Conventionally, cameras forming a camera array are arranged in a regular pattern such as a grid, a radial pattern, or a hemisphere. If cameras are arranged in a regular pattern, a problem occurs in that distance information about a single-color object having a size to some extent cannot be correctly acquired.
For example, as in FIG. 5A, it is assumed that a single-color bar X (for example, a handrail), which is a single-color object, is arranged parallel to the arrangement of a plurality of cameras. Images photographed by cameras A, B, and C are as illustrated in, for example, FIG. 5B. In FIG. 5B, black line portions correspond to image portions of the single-color object X in the respective images. In FIG. 5B, a, b, and c represent positions on an image corresponding to real-space positions (virtual object positions) a, b, and c illustrated in FIG. 5A.
As illustrated in FIGS. 5A and 5B, the pixel values in the images of the respective cameras corresponding to the positions a, b, and c are the same because the object X has a single color. Therefore, the pixel corresponding to the position b at which the single-color object X is located cannot be specified from the images of the respective cameras. Accordingly, the position b of the object X (object position) cannot be narrowed down to one point.
As illustrated in FIG. 5C, when an object having substantially uniform feature values is located in a plane that includes a straight line passing through two cameras, feature values substantially match at a plurality of virtual object positions. Thus, in principle, the object position cannot be specified using only two cameras. Therefore, it is necessary to add a further camera to narrow down virtual object positions. However, as described above, if another camera is added on the straight line passing through the two cameras, the object position cannot be specified. Thus, it is not effective to add a camera at such a position. Due to the physical constraint on the number of cameras arranged in a camera array, reducing as much as possible the number of cameras that do not contribute to an increase in the amount of information largely contributes to the production of a correct result of three-dimensional estimation and therefore the production of a high-quality virtual viewpoint image.
The present invention has been made to solve the above problem, and it is an object of the present invention to obtain a desired virtual viewpoint image or distance map by optimizing the position of an image capture unit.